{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "37ff3474",
   "metadata": {},
   "source": [
    "# 数据集处理\n",
    "在这一节，我们学习一下处理数据集常见的操作，MindSpore目前支持如数据清洗`shuffle`、数据分批`batch`、数据重复`repeat`、数据拼接`concat`等常用数据处理操作。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17cb55b1",
   "metadata": {},
   "source": [
    "## shuffle\n",
    "shuffle表示对数据进行清洗，即打乱数据集。当同一类别的数据连续输入神经网络时，可能会导致训练出的模型精度低，泛化能力弱。\n",
    "shuffle有一个`buffer_size`参数，设定的`buffer_size`越大，数据混洗程度越大，同时所消耗的时间、计算资源也更大。\n",
    "\n",
    "![shuffle](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r1.8/tutorials/source_zh_cn/advanced/dataset/images/op_shuffle.png)\n",
    "下面我们自定义一个数据集来模拟："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "19bbbe57",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[WARNING] ME(15596:4508,MainProcess):2022-10-21-23:40:08.338.187 [mindspore\\dataset\\engine\\datasets_user_defined.py:656] Python multiprocessing is not supported on Windows platform.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'data': Tensor(shape=[1], dtype=Int32, value= [0])}\n",
      "{'data': Tensor(shape=[1], dtype=Int32, value= [1])}\n",
      "{'data': Tensor(shape=[1], dtype=Int32, value= [2])}\n",
      "{'data': Tensor(shape=[1], dtype=Int32, value= [3])}\n",
      "{'data': Tensor(shape=[1], dtype=Int32, value= [4])}\n",
      "---------- after processing ----------\n",
      "{'data': Tensor(shape=[1], dtype=Int32, value= [0])}\n",
      "{'data': Tensor(shape=[1], dtype=Int32, value= [2])}\n",
      "{'data': Tensor(shape=[1], dtype=Int32, value= [3])}\n",
      "{'data': Tensor(shape=[1], dtype=Int32, value= [1])}\n",
      "{'data': Tensor(shape=[1], dtype=Int32, value= [4])}\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import mindspore.dataset as ds\n",
    "ds.config.set_seed(0)\n",
    "\n",
    "def generator():\n",
    "    \"\"\"定义生成数据集函数\"\"\"\n",
    "    for i in range(5):\n",
    "        yield (np.array([i]), )\n",
    "        \n",
    "# 创建数据集\n",
    "dataset = ds.GeneratorDataset(generator, [\"data\"])\n",
    "for data in dataset.create_dict_iterator():\n",
    "    print(data)\n",
    "\n",
    "print(\"---------- after processing ----------\")\n",
    "# 对比数据集\n",
    "dataset = dataset.shuffle(buffer_size=2)\n",
    "for data in dataset.create_dict_iterator():\n",
    "    print(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da565c03",
   "metadata": {},
   "source": [
    "## batch\n",
    "batch表示对数据进行分批输入神经网络。当数据集很大的时候，一次性全部输入神经网络可能会导致内存爆炸，故我们选择“小步快走”的方式输入数据，即进行分批输入。\n",
    "![batch](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r1.8/tutorials/source_zh_cn/advanced/dataset/images/op_batch.png)\n",
    "其中`batch_size`参数指定将数据分成多少个批次，`drop_remainder`参数表示是否遗弃最后剩下的不能分批的数据。\n",
    "\n",
    "下面我们自定义一个数据集来模拟："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "0ed78453",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[WARNING] ME(15596:4508,MainProcess):2022-10-21-23:52:17.762.03 [mindspore\\dataset\\engine\\datasets_user_defined.py:656] Python multiprocessing is not supported on Windows platform.\n",
      "[WARNING] ME(15596:4508,MainProcess):2022-10-21-23:52:17.895.94 [mindspore\\dataset\\engine\\datasets_user_defined.py:656] Python multiprocessing is not supported on Windows platform.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------- not drop remainder ----------\n",
      "{'data': Tensor(shape=[2, 1], dtype=Int32, value=\n",
      "[[0],\n",
      " [1]])}\n",
      "{'data': Tensor(shape=[2, 1], dtype=Int32, value=\n",
      "[[2],\n",
      " [3]])}\n",
      "{'data': Tensor(shape=[1, 1], dtype=Int32, value=\n",
      "[[4]])}\n",
      "---------- drop remainder ----------\n",
      "{'data': Tensor(shape=[2, 1], dtype=Int32, value=\n",
      "[[0],\n",
      " [1]])}\n",
      "{'data': Tensor(shape=[2, 1], dtype=Int32, value=\n",
      "[[2],\n",
      " [3]])}\n"
     ]
    }
   ],
   "source": [
    "print(\"---------- not drop remainder ----------\")\n",
    "dataset = ds.GeneratorDataset(generator, [\"data\"])\n",
    "dataset = dataset.batch(batch_size=2, drop_remainder=False)\n",
    "for data in dataset.create_dict_iterator():\n",
    "    print(data)\n",
    "    \n",
    "print(\"---------- drop remainder ----------\")\n",
    "dataset = ds.GeneratorDataset(generator, [\"data\"])\n",
    "dataset = dataset.batch(batch_size=2, drop_remainder=True)\n",
    "for data in dataset.create_dict_iterator():\n",
    "    print(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94e91900",
   "metadata": {},
   "source": [
    "## repeat\n",
    "repeat表示对数据进行重复，达到扩大数据集的目的，参数`count`表示重复的次数。\n",
    "\n",
    "注意：`repeat`和`batch`操作的先后顺序会影响训练batch的数量，建议将`repeat`置于`batch`之后。\n",
    "![repeat](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r1.8/tutorials/source_zh_cn/advanced/dataset/images/op_repeat.png)\n",
    "\n",
    "下面我们自定义一个数据集来模拟："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "c476a20f",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[WARNING] ME(15596:4508,MainProcess):2022-10-21-23:56:31.937.715 [mindspore\\dataset\\engine\\datasets_user_defined.py:656] Python multiprocessing is not supported on Windows platform.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'data': Tensor(shape=[1], dtype=Int32, value= [0])}\n",
      "{'data': Tensor(shape=[1], dtype=Int32, value= [1])}\n",
      "{'data': Tensor(shape=[1], dtype=Int32, value= [2])}\n",
      "{'data': Tensor(shape=[1], dtype=Int32, value= [3])}\n",
      "{'data': Tensor(shape=[1], dtype=Int32, value= [4])}\n",
      "---------- after processing ----------\n",
      "{'data': Tensor(shape=[1], dtype=Int32, value= [0])}\n",
      "{'data': Tensor(shape=[1], dtype=Int32, value= [1])}\n",
      "{'data': Tensor(shape=[1], dtype=Int32, value= [2])}\n",
      "{'data': Tensor(shape=[1], dtype=Int32, value= [3])}\n",
      "{'data': Tensor(shape=[1], dtype=Int32, value= [4])}\n",
      "{'data': Tensor(shape=[1], dtype=Int32, value= [0])}\n",
      "{'data': Tensor(shape=[1], dtype=Int32, value= [1])}\n",
      "{'data': Tensor(shape=[1], dtype=Int32, value= [2])}\n",
      "{'data': Tensor(shape=[1], dtype=Int32, value= [3])}\n",
      "{'data': Tensor(shape=[1], dtype=Int32, value= [4])}\n"
     ]
    }
   ],
   "source": [
    "dataset = ds.GeneratorDataset(generator, [\"data\"])\n",
    "for data in dataset.create_dict_iterator():\n",
    "    print(data)\n",
    "    \n",
    "print(\"---------- after processing ----------\")\n",
    "\n",
    "dataset = dataset.repeat(count=2)\n",
    "for data in dataset.create_dict_iterator():\n",
    "    print(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47d4a53d",
   "metadata": {},
   "source": [
    "## map\n",
    "`map接口`可以将指定的函数作用于数据集的指定列数据，我们可以使用其来自定义数据处理。其有两个入参，`operations`参数指定用于数据处理的函数，`input_columns`参数指定标签名。\n",
    "![map](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r1.8/tutorials/source_zh_cn/advanced/dataset/images/op_map.png)\n",
    "\n",
    "下面我们自定义一个数据集来模拟："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "1b0b07b7",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[WARNING] ME(15596:4508,MainProcess):2022-10-22-00:05:43.757.839 [mindspore\\dataset\\engine\\datasets_user_defined.py:656] Python multiprocessing is not supported on Windows platform.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'data': Tensor(shape=[1], dtype=Int32, value= [0])}\n",
      "{'data': Tensor(shape=[1], dtype=Int32, value= [1])}\n",
      "{'data': Tensor(shape=[1], dtype=Int32, value= [2])}\n",
      "{'data': Tensor(shape=[1], dtype=Int32, value= [3])}\n",
      "{'data': Tensor(shape=[1], dtype=Int32, value= [4])}\n",
      "---------- after processing ----------\n",
      "{'data': Tensor(shape=[1], dtype=Int32, value= [0])}\n",
      "{'data': Tensor(shape=[1], dtype=Int32, value= [2])}\n",
      "{'data': Tensor(shape=[1], dtype=Int32, value= [4])}\n",
      "{'data': Tensor(shape=[1], dtype=Int32, value= [6])}\n",
      "{'data': Tensor(shape=[1], dtype=Int32, value= [8])}\n"
     ]
    }
   ],
   "source": [
    "def operator(x):\n",
    "    \"\"\"定义数据处理函数\"\"\"\n",
    "    return 2*x\n",
    "\n",
    "dataset = ds.GeneratorDataset(generator, [\"data\"])\n",
    "for data in dataset.create_dict_iterator():\n",
    "    print(data)\n",
    "\n",
    "print(\"---------- after processing ----------\")\n",
    "# 调用map接口处理数据\n",
    "dataset = dataset.map(operations=operator, input_columns=[\"data\"])\n",
    "for data in dataset.create_dict_iterator():\n",
    "    print(data)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "mindspore",
   "language": "python",
   "name": "mindvision"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
